C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling beats Over-Sampling

This paper takes a new look at two sampling schemes commonly used to adapt machine algorithms to imbalanced classes and misclassification costs. It uses a performance analysis technique called cost curves to explore the interaction of over and under-sampling with the decision tree learner C4.5. C4.5 was chosen as, when combined with one of the sampling schemes, it is quickly becoming the community standard when evaluating new cost sensitive learning algorithms. This paper shows that using C4.5 with undersampling establishes a reasonable standard for algorithmic comparison. But it is recommended that the least cost classifier be part of that standard as it can be better than undersampling for relatively modest costs. Oversampling, however, shows little sensitivity, there is often little difference in performance when misclassification costs are changed.